Intelligent data propagation using performance monitoring

ABSTRACT

Exemplary methods, apparatuses, and systems that can intelligently copy data to a plurality of datastores using performance monitoring are described. In one embodiment, a shortest path tree determines the most efficient paths available for copying data from a source datastore to one or more destination datastores. During the copying of the data between a source datastore and the one or more destination datastores, a performance value of each of the datastores involved in the copying process is compared to a threshold. In response to determining that the performance value of a given source or destination datastore involved in the copying exceeds the threshold, the copying of the data to the corresponding destination datastore is suspended. An updated shortest path tree is determined to locate a more efficient path for copying data to the suspended destination datastore. Copying is resumed to the suspended destination datastore using the updated shortest path tree.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/080,718, filed Nov. 14, 2013, which is hereby incorporated byreference.

FIELD OF THE INVENTION

The various embodiments described herein relate generally to copyingdata in a highly distributed environment, and more particularly, topropagating copies of data to multiple host devices in a highlydistributed environment.

BACKGROUND OF THE INVENTION

A virtual machine clone is a copy of an existing virtual machine (VM).VMs can be cloned across highly distributed architectures. These highlydistributed environments include numerous host devices (e.g., rangingfrom hundreds to thousands of hosts) and corresponding datastores spreadacross geographically distinct locations. The speed at which VMs arecloned across a large, highly distributed environment, however, remainssub-optimal. For example, deploying a 200 cluster of VMs across 16 hostdevices by concurrently cloning a VM from a single template from asource datastore to all destination datastores takes on the order oftens of hours due in part to network and datastore bottlenecks.

Cloning VMs, or creating copies of other data, may be performed bycopying an entire file or by copying the file on a per-block basis(e.g., dividing the file to be copied into fragments and copying eachfragment from the source to the destination until the entire file hasbeen completely copied). The suboptimal copying speed described above,however, persists regardless of the manner in which the data is copiedfrom the source datastore to the destination datastores.

SUMMARY OF THE INVENTION

Exemplary methods, apparatuses, and systems intelligently copy data to aplurality of datastores using performance monitoring. In an exemplaryembodiment, a shortest path tree determines the most efficient pathsavailable for copying data from a source datastore to one or moredestination datastores. During the copying of the data between a sourcedatastore and the one or more destination datastores, a performancevalue of each of the datastores involved in the copying process iscompared to a threshold. In response to determining that the performancevalue of a given source or destination datastore involved in the copyingexceeds the threshold, the copying of the data to the correspondingdestination datastore is suspended. An updated shortest path tree isdetermined to locate a more efficient path for copying the data to thesuspended destination datastore. Copying is resumed to the suspendeddestination datastore using the updated shortest path tree.

In one embodiment, a distance value of a path between each datastore ina plurality of datastores is determined. The distance values are used toperform a graph cluster analysis. The graph cluster analysis createsclusters of datastores within close proximity to one another based onthe distance values. The source datastore is designated as the root ofthe shortest path tree and the one or more destination datastores aredesignated as the vertices of the tree. Each child vertex of the sourcedatastore is ordered in descending order according to a number of uniqueclusters to which descendants of the child vertex belong and the data iscopied from the source datastore to the one or more destinationdatastores in the descending order.

Other features and advantages will be apparent from the accompanyingdrawings and from the detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation in the figures of the accompanying drawings, in which likereferences indicate similar elements, and in which:

FIG. 1 illustrates, in block diagram form, an exemplary virtualdatacenter environment including one or more networked processingdevices to implement the intelligent propagation of copies of datathroughout a plurality of datastores;

FIG. 2A is a flow chart illustrating an exemplary method ofintelligently propagating copies of data throughout a plurality ofdatastores;

FIG. 2B is a flow chart illustrating additional features of the methodof intelligently propagating copies of data throughout a plurality ofdatastores;

FIG. 3 is a directed graph illustrating paths for copying data between aplurality of datastores according to an embodiment of the invention;

FIG. 4 illustrates a plurality of clusters of datastores createdaccording to an embodiment of the invention;

FIG. 5 is a shortest path tree used to propagate copies of dataaccording to an embodiment of the invention; and

FIG. 6 illustrates, in block diagram form, an exemplary processingsystem to implement the intelligent propagation of copies of datathroughout a plurality of datastores.

DETAILED DESCRIPTION

Embodiments described herein intelligently propagate copies of datathroughout a plurality of datastores in a highly distributedenvironment. Embodiments sort the datastores into different clustersbased upon determined distance values and prioritize branches of ashortest path tree that include diverse clusters. Additionally,embodiments suspend and resume copying between datastores based upon theperformance of each datastore during the copying process. As a result,copies of data can be more rapidly propagated throughout theenvironment.

FIG. 1 illustrates, in block diagram form, an exemplary virtualdatacenter environment 100 including one or more networked processingdevices implementing the intelligent propagation of copies of datathroughout a plurality of datastores. Server-based computing in avirtual datacenter environment allows client devices 105 to accesscentrally-managed user virtual desktops, such as those implemented byVMs 110, via network(s) 115 (e.g., a local area network or other privateor publically accessible wide area network, such as the Internet). Forexample, client device 105 and VM 110 use a desktop remoting protocolsuch as Remote Desktop Protocol (RDP), Virtual Network Computing (VNC),or Personal Computer over Internet Protocol (PCoIP) to remotelyaccess/provide remote access to a virtual desktop generated by VM 110.

The term “desktop,” as used herein, refers to a human interfaceenvironment through which users can launch, interact with, and manageapplications, settings, and/or data, etc. A virtual desktop is a virtualcomputing system that operates as a desktop or workstation computer withwhich an end user can interact using a desktop remoting protocol and thesoftware and/or hardware of client device 105. A virtualized desktop mayexport user interfaces, e.g., keyboard and mouse input, audio and visualoutput, to client device 105 from VM 110 running remotely (in thedatacenter) or locally on client device 105, or, in some cases, using ahybrid. Client device 105 transmits user input, e.g., keyboard or mouseinput, to VM 110 to be processed, and receives display and other data,such as sound, from VM 110 for presentation to a user of client device105. Client device 105 presents a bitmap representation of the desktopgenerated by VM 110. Input to and output from the virtualized desktopare reflected in the bitmap representation on client device 105 as itoccurs on the virtualized desktop within VM 110.

VMs 110 are complete computation environments, containing virtualequivalents of the hardware and system software components of a physicalsystem and are typically implemented by an extensive virtualizationinfrastructure, which includes a variety of software and hardwarecomponents. A virtualization software layer 120 (e.g., a hypervisor)running on hardware 125 of physical computer 135 manages one or more VMs110. Virtualization software layer 120 manages physical resources, e.g.,hardware 125, as well as maintains virtual-to-physical hardwaremappings.

Additionally, virtualization software layer 120 includes propagationagent 180. In one embodiment, propagation agent 180 performs at least aportion of method 200 described below. For example, propagation agent180 facilitates the transfer of data from a source datastore to adestination datastore. Additionally, a lead propagation agent 180 maycommunicate with multiple propagation agents 180 to perform method 200.Alternatively, software interface 145 of virtual machine managementserver (VMMS) 140 includes one or more propagation managers 185 toperform at least a portion of method 200, as described below. While leadpropagation agent 180 and propagation manager 185 have been described asalternatives to one another, in one embodiment, lead propagation agent180 and propagation manager 185 are used together to perform method 200described below. For example, one or both of lead propagation agent andpropagation manager 185 may request and/or receive performance data,propagation status, etc. from propagation agent(s) 180; determineclusters, shortest path trees, etc. and direct host devices 135 and/ordatastores 150 to initiate, pause, or resume the propagation of data.For the sake of explanation herein, however, reference will simply bemade to propagation manager 185.

In yet another embodiment, datastores 150 include a propagation agent(not shown) to perform one or more of the functionalities described withreference to propagation agent 180 and method 200. In one embodiment,each datastore 150 is a single storage device that stores data for oneor more VMs 110. Alternatively, each datastore 150 represents a diskarray or other collection data stores. For example, each datastore 150may implement a redundant array of independent disks (RAID) algorithm tocombine multiple drive components into a logical unit and divide and/orreplicate data among multiple physical drives. In one embodiment,datastore 150 includes a processing device or other storage controller(not shown).

Remote access to virtualized desktops is generally provided to clientdevices 105 through virtual infrastructure management (VIM) server 130.In one embodiment, the VIM server 130 provides virtual desktop access tothe client devices 105. Additionally, or alternatively, VIM server 130orchestrates the provisioning of software defined datacenter services.Exemplary services include virtualized compute, networking, storage, andsecurity services that implement the infrastructure of virtualdatacenter 100. VIM server 130 manages the corresponding VMs 110 throughcommunications with software interface 145 of VMMS 140. VMMS 140 isresponsible for provisioning, cloning, and maintaining the multitude ofVMs 110 implemented across one or more physical computers 135 as well asdatastores 150 utilized by VMs 110.

VIM server 130 may be a physical computer system or a virtual machinethat runs infrastructure management software 155. Infrastructuremanagement software 155 within VIM server 130 manages pools of computerresources to run VMs 110 on a set of clusters typically includingmultiple servers/physical computers 135 with central/graphics processingunits (CPU's and/or GPU's), memory, and communications hardware.Infrastructure management software 155 includes one or more modules,including administrative console 160, inventory manager 165, connectionbroker 170, and pool manager 175.

Administrative console 160 provides a remotely accessible user interfaceto administrator device 106 to manage the configuration of VMs 110within the virtual datacenter 100. In one embodiment, administrativeconsole 160 exports a graphical user interface via hypertext transferprotocol (HTTP) to be accessed by a web browser. Alternatively, acommand-line interface or a rich client is provided to administratordevice 106, which includes a web browser to access the administrativeconsole. Administrative console 160 allows administrator device 106 toperform a plurality of functions, such as: cloning VMs, creating desktoppools, associating desktop pools with VIM server 130, associating adesktop pool with a master image, defining VM state policies, setting upcustom messages to be sent to users (e.g., when the desktops are putinto maintenance mode for weekly patches, hotfixes, etc.), settingstorage overcommit settings for different datastores 150 used by desktoppools, performing rebalance operations on a set of virtual desktops,configuring firewall and other networking/security services, addingservers, isolating multi-tenant organizations, creating datacenterservice templates, defining role-based access, monitoring processor,memory, and storage usage/allocation, pooling resources, and othersimilar operations. For example, administrative console 160 may provideadmin device 106 access to initiate the cloning of a VM to a pluralityof datastores 150 as described herein.

Inventory manager 165 handles the events received from VMs 110. Forexample, when a user of a virtual desktop logs off or disconnects, asoftware agent running within a VM 110 sends a notification to inventorymanager 165. Inventory manager 165 determines an effective policy thatapplies to the desktop and performs a maintenance operation, ifrequired. Inventory manager 165 may also use timer-based events toschedule effective maintenance operations.

Connection broker 170 allows a remote user or administrator, through aclient or administrator device 105, 106, to select a type of virtualdesktop and initiate a virtual desktop session or to access an existingconnection to a virtual desktop. In one embodiment, connection broker170 connects to VM 110 to access or update policies associated with VM110 (e.g., to enable or prohibit virtual desktop shadowing).

Pool manager 175 manages the lifecycle of virtual desktops. Virtualdesktops in a pool are grouped together based on similar softwarerequirements. Desktop Administrators create logical desktops groups(desktop pools) that are provisioned typically from the same base image.For example, a desktop pool may include cloned VMs 110 that run the sameset of software applications and run the same operating system.

In general, VMMS 140 provides software interface 145 to enable anadministrator or other entity, such as inventory manager 165, connectionbroker 170, and pool manager 175, to access and manage VMs 110 asdescribed above.

VMMS 140 also supports operations for the discovery of compute, storage,and network resources; creation of logical compute pools by providingfeatures such as automatic CPU and memory load balancing;provisioning/creation of one or more virtual disks, on local or sharedstorage-clones with full or sparse disks; creation of cloned VMs; andpower operations on virtual machines (power on, power-off, suspend,resume, checkpoint, etc.).

While illustrated as separate components, in one embodiment, a singleserver or other networked processing device implements VIM server 130and VMMS 140. For example, VIM server 130 and VMMS 140 may beimplemented by one or more VMs 110.

FIG. 2A and FIG. 2B illustrate method 200 of intelligently propagatingcopies of data throughout a plurality of datastores. For example, thedata is copied and propagated throughout datastores 150 in FIG. 1described above. The data to be copied can be a VM, a snapshot of a VM,the files that make up a VM (e.g., VM log files, virtual disk files, VMredo-log files, VM paging files, VM configuration files, etc.) or anyother type of files (related to/unrelated to a VM) that can bepropagated across a highly distributed environment. Additionally, thedata to be copied can be an entire file or a fragment of a file (e.g.,fragments of files used to transfer data between peers in a peer-to-peerprotocol such as BitTorrent). While described with reference to virtualdatacenter environment 100, method 200 may also be utilized to copy datain other network environments.

In FIG. 2A, at block 202, propagation manager 185 determines a distancevalue between each datastore 150. The distance value of the path betweeneach datastore 150 is based upon a network performance measurementbetween datastores 150. For example, the network performance measurementcan be based on measurements of network or input/output (I/O) latency.

Network latency refers to a measurement of the amount of time it takesfor a packet of data to get from one designated point to another in thenetwork. Examples of measurements to show network latency include, butare not limited to, throughput, bandwidth-delay, bit-error rate,round-trip time (e.g., as determined by a ping test), or any combinationthereof. Alternatively, the network latency can be based on measurementsof hop counts between each datastore 150.

I/O latency refers to the time in which it takes a datastore to completea task. Processing unit contention and network bandwidth contention maycontribute to I/O latency. There are various ways in which the I/Olatency of each datastore 150 could be measured. Examples of thesemeasurements include, but are not limited to, the time it takes aparticular datastore 150 to store or retrieve a file of a given size andthe time it takes a particular datastore 150 to copy a file of a givensize to or from other datastores.

In one embodiment, propagation manager 185 receives network performancedata from propagation agents 180. For example, as each host device 135or datastore 150 determines a corresponding network latency (withrespect to each other host device 135 or each other datastore 150) orI/O latency, a corresponding propagation agent 180 provides propagationmanager 185 with the corresponding value(s). Propagation manager 185maintains a table or other data structure to store the received networkperformance data values.

At block 204, propagation manager 185 optionally determines a datastoreperformance value for each datastore 150. The datastore performancevalue can be based on the performance capabilities of each datastore150. For example, and in one embodiment, the datastore performance valuecan be based on measurements of resource utilization or I/O speed ofeach datastore 150.

Resource utilization refers to a measure of how much data storage isavailable and/or how well the available data storage space in adatastore is used. Resource utilization can also be a measure of theavailability of a datastore to process pending tasks. For example,resource utilization may be based upon a number of pending tasks at agiven time, an average number of pending tasks for the resource overtime, a percentage of time the resource is occupied processing tasks,etc. There are a number of variables that can be used, alone or incombination, to determine the resource utilization of each datastore150. The relative priorities assigned to each variable can also affectthe resource utilization figure. Examples of variables that can be usedto determine the resource utilization include, but are not limited to,the total available storage space in each datastore 150, thetime-averaged percentage of storage space used in each datastore 150,the percentage of storage space used at times of peak demand in eachdatastore 150, the number of pending tasks of each datastore 150, theaverage number of pending tasks of each datastore 150, the percentage oftime each datastore 150 spends processing tasks, and the average size offiles stored on or retrieved from each datastore 150.

The I/O speed refers to the time it takes a datastore to complete a taskdetermined principally by the period spent waiting for I/O operations tobe completed. There are various ways in which the I/O speed of datastore150 could be measured. Examples of these measurements include, but arenot limited to, the average time required to store or retrieve a file ofa given size for each datastore 150, the time during peak-demand periodsrequired to store or retrieve a file of a given size in each datastore150, and the time required to copy a file of a given size from or toeach datastore 150.

In one embodiment, both the resource utilization and the I/O speedmeasurements are based on current or historical values. Furthermore,resource utilization and the I/O speed measurements can be usedseparately or in combination.

In one embodiment, the datastore performance value is based upon otherhardware characteristics for a given datastore 150. For example, thedatastore performance value may be based upon processing unit speed,disk speed, etc.

At block 206, propagation manager 185 optionally modifies the distancevalues between each datastore 150 using the datastore performance valuesof each datastore 150. For example, the datastore performance value mayserve as a multiplier or another weighting value for a correspondingdistance value. The modification is performed to create a modifieddistance value to represent the relative distances between eachdatastore 150 in combination with the performance capabilities of eachdatastore 150. An exemplary representation of the result of theprocesses of blocks 202-206 of method 200 is found in FIG. 3.

FIG. 3 is a directed graph 300 illustrating paths for copying databetween datastores 150. Directed graph 300 shows the relative positionsof datastores 150 based on the distance values or modified distancevalues, e.g., as determined in blocks 202-206 of method 200. A subset ofdatastores 150 is represented in directed graph 300 as one of nodes301A-N. A larger or smaller number of datastores may be included indirected graph 300. Between each node is an edge 303, which representsthe distance values or modified distance values between the datastores150. The arrows indicate exemplary directions in which data can be sent.For example, the arrows at node 301B indicate that data can be sent fromnode 301B to itself (e.g., a VM may be cloned within a datastore 150 foruse by the same or a different host device that also accesses thedatastore 150), from node 301A to node 301B, and from node 301B to node301A.

Returning to FIG. 2A, at block 208, propagation manager 185 performs agraph cluster analysis using the distance values or the modifieddistance values to create a plurality of clusters. Propagation manager185 may use any of a number of clustering algorithms to create theplurality of clusters. Each cluster includes one or more datastores 150within close proximity to one another based on the distance values ormodified distance values (e.g., as determined in blocks 202-206).

FIG. 4 is an exemplary illustration of the result of the graph clusteranalysis. Clusters 401A-D are made up of a subset of datastores 150 thatare in close proximity to one another based on the distance values ormodified distance values, including nodes 301A-N described above. Forexample, nodes 301A, 301B, 301F in cluster 401A represent datastores 150that are in close proximity to one another. As described further herein,node 301B represents the source datastore, which is the one ofdatastores 150 that stores the data to be copied to the other datastores150. The remainder of datastores 150 that receive the copied data arereferred to herein as destination datastores. While clusters 401B-D arenot illustrated or described as containing any source datastores,clusters 401B-D can contain source datastores. Furthermore, there can bemore than one source datastore in one or more of clusters 401A-D.

Returning to FIG. 2A, at block 210, propagation manager 185 determines ashortest path tree to copy data from a source datastore (which is placedat the root of the shortest path tree) to the destination datastores(which are represented as vertices on the shortest path tree). Forexample, propagation manager 185 utilizes Dijkstra's algorithm (using asingle or multiple sources) or another shortest path algorithm to selectedge(s) 303 in directed graph 300 to copy the data from node 301B tonodes 301A and 301C-N.

FIG. 5 illustrates shortest path tree 500 used to determine the pathsalong which copies of data will be transmitted. For example, data may becopied from 301B to 301G and from 301G to 301E. Alternatively, the datamay be copied from 301B to 301E via 301C and 301D. Given thatpropagation manager 185 determined the path including 301G to be theshortest, shortest path tree 500 includes that path.

Nodes 301A-N are arranged in a hierarchy based on the shortest path ofcopying to each node. For example, as a result of the shortest path treecalculation, source node 301B will copy data directly to nodes 301A,301G, and 301C. Nodes 301A, 301G, and 301C are, therefore, in the firstlevel of the hierarchy. Similarly, nodes 301D-F are in the second levelof the hierarchy, nodes 301H-L are in the third level of the hierarchy,and nodes 301M-N are in the fourth level of the hierarchy.

Returning to FIG. 2A, at block 212, propagation manager 185 determinesan ordering for each child vertex of the root of the shortest path treein descending order. In one embodiment, propagation manager 185 placesthe child vertices (of the source datastore) in descending orderaccording to a number of unique clusters to which the descendants of thechild vertex belong. For example, the child vertices of source node 301Bare illustrated in the first level of hierarchy in FIG. 5. Node 301A hasno descendants. Node 301G has five descendants: two child vertices, 301Fand 301E, one grandchild vertex, 301H, and two great grandchildvertices, 301M and 301N. Node 301C has five descendants: one childvertex, 301D, and four grandchild vertices, 301I-L. Referring to FIG. 4,the descendants of node 301 G represent three different clusters—node301F is in cluster 401A, nodes 301E and 301H are in cluster 401 C, andnodes 301M and 301N are in cluster 401D. The descendants of node 301Crepresent two different clusters—node 301K is in cluster 401C, and nodes301D, 301I, 301J, and 301L are in cluster 401B. In this example, thenode with the highest number of descendants that belong to uniqueclusters, is ranked first. Node 301C, which has only five descendants intwo unique clusters will be ranked after node 301G, which has fivedescendants in three unique clusters. Node 301A which has no descendantswill be ranked after node 301C.

The ordering of each child vertex can be additionally or alternativelydetermined by basing the order upon the number of children/descendantsof each child vertex, the resource utilization of each child vertex'scorresponding datastore, the I/O latency of each child vertex'scorresponding datastore, the network latency of each child vertex'scorresponding datastore (with respect to each other host device or eachother datastore), or any combination thereof. For example, if nodes 301Gand 301C were to both have descendants that belong to three uniqueclusters, but node 301C had a greater number of descendants, propagationmanager 185 may rank node 301C higher in the order than node 301G.

In another embodiment, the ordering of the each child vertex can bedetermined by the number of descendants in a particular generationbeneath the child vertex that belong to unique clusters. For instance,using the same example described above, the ordering of the childvertices of Node 301B could be based on the highest number ofgrandchildren of each child vertex that belong to unique clusters. Thechild vertices of source node 301B and their corresponding grandchildvertices are illustrated in the first and third levels of hierarchy inFIG. 5, respectively. Node 301A has no grandchildren. Node 301C has fourgrandchildren, 301I-L. Node 301G has one grandchild vertex, 301H.Referring to FIG. 4, the grandchild of node 301G represents only onecluster—node 301H is in cluster 401C. The grandchildren of node 301Crepresent two different clusters—node 301K is in cluster 401C, and nodes301I, 301J, and 301L are in cluster 401B. In this example, node 301C,which has the highest number of grandchildren who belong to uniqueclusters, is ranked first. Node 301G, which has only one grandchild inone unique cluster will be ranked after node 301C. Node 301A which hasno descendants will be ranked after node 301G. In another embodiment,the ordering of the each child vertex can be determined using the numberof unique clusters represented by descendants within a threshold numberof generations. For example, propagation manager 185 may determine thenumber of unique clusters represented by descendants within one, two,three, or another number of generations beneath each child vertex in thefirst hierarchy level. As stated above, the ordering of each childvertex can be further refined by basing the order upon the number ofchildren/descendants of each child vertex, the resource utilization ofeach child vertex's corresponding datastore, the I/O latency of eachchild vertex's corresponding datastore, network latency betweendatastores, or any combination thereof.

Returning to FIG. 2A, at block 214, propagation manager 185 copies thedata from the source datastore to a destination datastore represented asa child vertex of the source datastore, e.g., based upon the determinedordering. In one embodiment, the copying of the data is performed bypropagation agent(s) 180. For example, a source propagation agent 180requests data from source datastore 150 and transmits the data to adestination propagation agent 180 to store with destination datastore150. In another embodiment, the copying of the data is performed by bothpropagation manager 185 and propagation agent(s) 180.

Propagation manager 185 determines, at block 216, if the copying of thedata from the source datastore to one or more of the destinationdatastores has completed within a threshold period of time. If thecopying has not been completed, method 200 moves to block 222 of FIG. 2B(described below). If the copying has been completed, at block 218,propagation manager 185 determines whether a threshold number of copieshas been reached. If the threshold number of copies has been reached,method 200 returns to block 210 and propagation manager 185 determinesan updated shortest path, as described above. The threshold number ofcopies is a predetermined number of copies used by propagation manager185 and/or propagation agent(s) 180 to refine the selection of paths forcopying the data. For example, a default value or administrator setvalue for the threshold may be ten. Once the data has been copied(directly or indirectly) from a source datastore to ten destinationdatastores, method 200 returns to block 210 and propagation manager 185determines an updated shortest path.

In one embodiment, the updated shortest path is determined using one ormore different source datastores 150 (in addition to or as analternative to the original source datastore 150). For example, with thethreshold number of copies having been made, additional sourcedatastores 150 are now available to propagate additional copies. As aresult, propagation manager 185 may use a multi-source shortest pathtree algorithm to determine the updated shortest path tree(s).

If the threshold number of copies has not been reached, method 200proceeds to block 220, in FIG. 2A. At block 220, propagation manager 185promotes the child vertex of the source datastore to a new sourcedatastore. For example, in FIG. 5, the data is copied from sourcedatastore 301B to the child vertex of the source datastore, datastore301G. The child vertex 301G becomes the new source datastore aftercompleting the copying of the data from node 301B.

After the promotion, propagation manager 185 returns to block 212 andresumes method 200 as described above. For example, with the promotionof datastore 301G to the new source datastore, datastores represented bychild vertices of datastore 301G (datastores 301E and 301F) are orderedand data is copied to each of them in the descending order.

As described above, if the copying performed by propagation manager 185at block 216 in FIG. 2A has not been completed within a threshold periodof time, method 200 proceeds to block 222 of FIG. 2B. At block 222,propagation manager 185 determines whether one or more datastores 150has exceeded a performance threshold during the copying of the data tothe datastore or during the copying of the data from the datastore. Theperformance threshold is a predetermined value based on the measurementsof resource utilization or I/O latency of datastores 150, both of whichare described above. For example, the performance threshold may be setto an expected, average, or above-average value for resource utilizationor I/O latency and propagation manager 185 (or propagation agent 180)determines that the current resource utilization or I/O latency exceedsthe performance threshold during the copying of data.

If the performance threshold has not been exceeded, then method 200returns to block 214 and continues with method 200 as described above.If, however, the one or more datastores 150 involved in the copyingprocess exceed the performance threshold, at block 224, propagationmanager 185 suspends the copying of the data to the datastore 150involved in the copying process. For example, propagation agent(s) 180sets the threshold to a value of 75% of the resource utilization. If adatastore 150 involved in the copying process reports to propagationmanager 185 via propagation agent(s) 180 that the datastore 150 exceeds75% resource utilization prior to or during the copying of data, method200 proceeds to block 224. At block 224, propagation manager 185suspends the copying of the file to the datastore 150 in response tothat datastore 150 exceeding the threshold.

After the copying is suspended at block 224 of FIG. 2B, method 200proceeds to block 218 and propagation manager 185 determines if athreshold number of copies has been reached, as described above. If thethreshold number of copies has not been reached, copying of datacontinues (for datastores other than the suspended datastore). If thethreshold number of copies has been reached, propagation manager 185determines an updated shortest path tree and propagation manager 185and/or propagation agent(s) 180 resume copying the data to the datastorethat exceeded the threshold using the updated shortest path tree.

FIG. 6 illustrates, in block diagram form, processing system 600 toimplement the intelligent propagation of copies of data throughout aplurality of datastores. Data processing system 600 includes one or moremicroprocessors 605 and connected system components (e.g., multipleconnected chips). Alternatively, data processing system 600 is a systemon a chip.

Data processing system 600 includes memory 610, which is coupled tomicroprocessor(s) 605. Memory 610 may be used for storing data,metadata, and programs for execution by the microprocessor(s) 605.Memory 610 may include one or more of volatile and non-volatilememories, such as Random Access Memory (“RAM”), Read Only Memory(“ROM”), a solid state disk (“SSD”), Flash, Phase Change Memory (“PCM”),or other types of data storage. Memory 610 may be internal ordistributed memory.

Data processing system 600 includes network and port interfaces 615,such as a port, connector for a dock, or a connector for a USBinterface, FireWire, Thunderbolt, Ethernet, Fibre Channel, etc. toconnect the system 600 with another device, external component, or anetwork. Exemplary network and port interfaces 615 also include wirelesstransceivers, such as an IEEE 802.11 transceiver, an infraredtransceiver, a Bluetooth transceiver, a wireless cellular telephonytransceiver (e.g., 2G, 3G, 4G, etc.), or another wireless protocol toconnect data processing system 600 with another device, externalcomponent, or a network and receive stored instructions, data, tokens,etc.

Data processing system 600 also includes display controller and displaydevice 620 and one or more input or output (“I/O”) devices andinterfaces 625. Display controller and display device 620 provides avisual user interface for the user. I/O devices 625 allow a user toprovide input to, receive output from, and otherwise transfer data toand from the system. I/O devices 625 may include a mouse, keypad or akeyboard, a touch panel or a multi-touch input panel, camera, opticalscanner, audio input/output (e.g., microphone and/or a speaker), otherknown I/O devices or a combination of such I/O devices.

It will be appreciated that one or more buses, may be used tointerconnect the various components shown in FIG. 6.

Data processing system 600 is an exemplary representation of one or moreof the client device 105, administrator device 106, VIM server 130,physical computer 135, and VMMS 140 described above. Data processingsystem 600 may be a personal computer, tablet-style device, a personaldigital assistant (PDA), a cellular telephone with PDA-likefunctionality, a Wi-Fi based telephone, a handheld computer whichincludes a cellular telephone, a media player, an entertainment system,or devices which combine aspects or functions of these devices, such asa media player combined with a PDA and a cellular telephone in onedevice. In other embodiments, data processing system 600 may be anetwork computer, server, or an embedded processing device withinanother device or consumer electronic product. As used herein, the termscomputer, device, system, processing system, processing device, and“apparatus comprising a processing device” may be used interchangeablywith data processing system 600 and include the above-listed exemplaryembodiments.

It will be appreciated that additional components, not shown, may alsobe part of data processing system 600, and, in certain embodiments,fewer components than that shown in FIG. 6 may also be used in dataprocessing system 600. It will be apparent from this description thataspects of the inventions may be embodied, at least in part, insoftware. That is, the computer-implemented method 200 may be carriedout in a computer system or other data processing system 600 in responseto its processor or processing system 605 executing sequences ofinstructions contained in a memory, such as memory 610 or othernon-transitory machine-readable storage medium. The software may furtherbe transmitted or received over a network (not shown) via networkinterface device 615. In various embodiments, hardwired circuitry may beused in combination with the software instructions to implement thepresent embodiments. Thus, the techniques are not limited to anyspecific combination of hardware circuitry and software, or to anyparticular source for the instructions executed by data processingsystem 600.

An article of manufacture may be used to store program code providing atleast some of the functionality of the embodiments described above.Additionally, an article of manufacture may be used to store programcode created using at least some of the functionality of the embodimentsdescribed above. An article of manufacture that stores program code maybe embodied as, but is not limited to, one or more memories (e.g., oneor more flash memories, random access memories—static, dynamic, orother), optical disks, CD-ROMs, DVD-ROMs, EPROMs, EEPROMs, magnetic oroptical cards or other type of non-transitory machine-readable mediasuitable for storing electronic instructions. Additionally, embodimentsof the invention may be implemented in, but not limited to, hardware orfirmware utilizing an FPGA, ASIC, a processor, a computer, or a computersystem including a network. Modules and components of hardware orsoftware implementations can be divided or combined withoutsignificantly altering embodiments of the invention.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. Various embodimentsand aspects of the invention(s) are described with reference to detailsdiscussed herein, and the accompanying drawings illustrate the variousembodiments. The description above and drawings are illustrative of theinvention and are not to be construed as limiting the invention.References in the specification to “one embodiment,” “an embodiment,”“an exemplary embodiment,” etc., indicate that the embodiment describedmay include a particular feature, structure, or characteristic, but notevery embodiment may necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Furthermore, when a particularfeature, structure, or characteristic is described in connection with anembodiment, such feature, structure, or characteristic may beimplemented in connection with other embodiments whether or notexplicitly described. Blocks with dashed borders (e.g., large dashes,small dashes, dot-dash, dots) are used herein to illustrate optionaloperations that add additional features to embodiments of the invention.However, such notation should not be taken to mean that these are theonly options or optional operations, and/or that blocks with solidborders are not optional in certain embodiments of the invention.Numerous specific details are described to provide a thoroughunderstanding of various embodiments of the present invention. However,in certain instances, well-known or conventional details are notdescribed in order to provide a concise discussion of embodiments of thepresent inventions.

It will be evident that various modifications may be made theretowithout departing from the broader spirit and scope of the invention asset forth in the following claims. For example, the methods describedherein may be performed with fewer or more features/blocks or thefeatures/blocks may be performed in differing orders. Additionally, themethods described herein may be repeated or performed in parallel withone another or in parallel with different instances of the same orsimilar methods.

What is claimed is:
 1. A computer-implemented method comprising: copyingdata to a plurality of datastores, wherein the copying of the dataincludes determining a path from a source datastore to a remainder ofthe plurality of datastores; determining that one of the plurality ofdatastores has exceeded a performance threshold during the copying ofthe data to the datastore or during the copying of the data from thedatastore; suspending the copying of the data to or from the datastorein response to the datastore exceeding the performance threshold;determining an updated path from a source datastore to the datastore;and resuming the copying of the data to the datastore using the updatedpath.
 2. The computer-implemented method of claim 1, further comprising:generating a snapshot of a virtual machine from the source datastore;and running a copy of the virtual machine on a host device using thesnapshot while the copying is performed, wherein the data that is copiedfrom the source datastore to one or more of the plurality of datastoresis the virtual machine.
 3. The computer-implemented method of claim 1,further comprising: determining a distance value of a path between eachof the plurality of datastores based upon a network performancemeasurement between the datastores; and performing a graph clusteranalysis using the distance values to create a plurality of clusters,wherein each cluster includes one or more of the datastores within closeproximity to one another based upon the distance values.
 4. Thecomputer-implemented method of claim 3, further comprising: determininga performance value for each datastore, wherein the datastoreperformance value is based on resource utilization, network latency, orI/O latency of the corresponding datastore; and modifying the distancevalues using corresponding datastore performance values of one or moredatastores in the corresponding path, wherein the graph cluster analysisuses the modified distance values.
 5. The computer-implemented method ofclaim 3, wherein the determination of the path or the updated pathincludes representing each datastore as a vertex in a shortest pathtree, the source datastore as a root of the shortest path tree, andordering each child vertex of the root in descending order according toa number of unique clusters to which descendants of the child vertexbelong.
 6. The computer-implemented method of claim 5, wherein theresuming the copying of the data on the datastore using the updatedshortest path tree is based on the ordering each child vertex of theroot in descending order according to a number of unique clusters towhich descendants of the child vertex belong.
 7. Thecomputer-implemented method of claim 5, wherein the ordering each childvertex in the shortest path tree or the updated shortest path tree indescending order is further based upon a number of children of eachchild vertex, resource utilization of each child vertex's correspondingdatastore, I/O latency of each child vertex's corresponding datastore,or network latency of each child vertex's corresponding datastore. 8.The computer-implemented method of claim 5, wherein copying the datacomprises: copying the data from the source datastore to a datastorerepresented by a first child vertex of the source datastore; promotingthe first child vertex to a new source datastore upon completion of thecopying of the data to the first child vertex; ordering each childvertex of the new source datastore in descending order according to anumber of unique clusters to which descendants of the child vertex ofthe new source datastore belong; and copying the data from the newsource datastore to one or more of the plurality of datastores in thedescending order of the child vertices of the new source datastore. 9.The computer-implemented method of claim 5 wherein the determining anupdated shortest path tree is in response to a threshold number of timesthe data is copied to the plurality of datastores, the updated shortestpath tree including an updated source datastore that is one of thesource datastore and another datastore storing a copy of the data, themethod further comprising: ordering each child vertex of the updatedsource datastore in descending order according to a number of uniqueclusters to which descendants of the child vertex of the updated sourcedatastore belong; and copying the data from the updated source datastoreto one or more of the plurality of datastores in the descending order ofthe child vertices of the updated source datastore.
 10. A non-transitorycomputer-readable medium storing instructions, which when executed by aprocessing device, cause the processing device to perform a methodcomprising: copying data to a plurality of datastores, wherein thecopying of the data includes determining a path from a source datastoreto a remainder of the plurality of datastores; determining that one ofthe plurality of datastores has exceeded a performance threshold duringthe copying of the data to the datastore or during the copying of thedata from the datastore; suspending the copying of the data to or fromthe datastore in response to the datastore exceeding the performancethreshold; determining an updated path from a source datastore to thedatastore; and resuming the copying of the data to the datastore usingthe updated path.
 11. The non-transitory computer-readable medium ofclaim 10, the method further comprising: generating a snapshot of avirtual machine from the source datastore; and running a copy of thevirtual machine on a host device using the snapshot while the copying isperformed, wherein the data that is copied from the source datastore toone or more of the plurality of datastores is the virtual machine. 12.The non-transitory computer-readable medium of claim 10, the methodfurther comprising: determining a distance value of a path between eachof the plurality of datastores based upon a network performancemeasurement between the datastores; and performing a graph clusteranalysis using the distance values to create a plurality of clusters,wherein each cluster includes one or more of the datastores within closeproximity to one another based upon the distance values.
 13. Thenon-transitory computer-readable medium of claim 12, the method furthercomprising: determining a performance value for each datastore, whereinthe datastore performance value is based on resource utilization,network latency, or I/O latency of the corresponding datastore; andmodifying the distance values using corresponding datastore performancevalues of one or more datastores in the corresponding path, wherein thegraph cluster analysis uses the modified distance values.
 14. Thenon-transitory computer-readable medium of claim 12, wherein thedetermination of the path or the updated path includes representing eachdatastore as a vertex in a shortest path tree, the source datastore as aroot of the shortest path tree, and ordering each child vertex of theroot in descending order according to a number of unique clusters towhich descendants of the child vertex belong.
 15. The non-transitorycomputer-readable medium of claim 14, wherein the resuming the copyingof the data on the datastore using the updated shortest path tree isbased on the ordering each child vertex of the root in descending orderaccording to a number of unique clusters to which descendants of thechild vertex belong.
 16. The non-transitory computer-readable medium ofclaim 14, wherein the ordering each child vertex in the shortest pathtree or the updated shortest path in descending order is further basedupon a number of children of each child vertex, resource utilization ofeach child vertex's corresponding datastore, I/O latency of each childvertex's corresponding datastore, or network latency of each childvertex's corresponding datastore.
 17. The non-transitorycomputer-readable medium of claim 14, wherein copying the datacomprises: copying the data from the source datastore to a datastorerepresented by a first child vertex of the source datastore; promotingthe first child vertex to a new source datastore upon completion of thecopying of the data to the first child vertex; ordering each childvertex of the new source datastore in descending order according to anumber of unique clusters to which descendants of the child vertex ofthe new source datastore belong; and copying the data from the newsource datastore to one or more of the plurality of datastores in thedescending order of the child vertices of the new source datastore. 18.The computer implemented method of claim 14, wherein the determining anupdated shortest path tree is in response to a threshold number of timesthe data is copied to the plurality of datastores, the updated shortestpath tree including an updated source datastore that is one of thesource datastore and another datastore storing a copy of the data, themethod further comprising: ordering each child vertex of the updatedsource datastore in descending order according to a number of uniqueclusters to which descendants of the child vertex of the updated sourcedatastore belong; and copying the data from the updated source datastoreto one or more of the plurality of datastores in the descending order ofthe child vertices of the updated source datastore.
 19. An apparatuscomprising: a processing device, wherein the processing device executesinstructions that cause the apparatus to perform a method comprising:copying data to a plurality of datastores, wherein the copying of thedata includes determining a path from a source datastore to a remainderof the plurality of datastores; determining that one of the plurality ofdatastores has exceeded a performance threshold during the copying ofthe data to the datastore or during the copying of the data from thedatastore; suspending the copying of the data to or from the datastorein response to the datastore exceeding the performance threshold;determining an updated path from a source datastore to the datastore;and resuming the copying of the data to the datastore using the updatedpath.
 20. The apparatus of claim 19, the method further comprising:determining a distance value of a path between each of the plurality ofdatastores based upon a network performance measurement between thedatastores; and performing a graph cluster analysis using the distancevalues to create a plurality of clusters, wherein each cluster includesone or more of the datastores within close proximity to one anotherbased upon the distance values, wherein the determination of theshortest path tree or the updated shortest path tree includesrepresenting each datastore as a vertex in the shortest path tree, thesource datastore as a root of the shortest path tree, and ordering eachchild vertex of the root in descending order according to a number ofunique clusters to which descendants of the child vertex belong.